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(54) An apparatus for editing moving picture having a related information thereof, an editing 
method and recording medium for storing procedures in the same method 

(57) An apparatus for editing moving picture having 

a related information thereof comprising; input means FIG. 1 

for inputting said moving picture and said related infor- 
mation, said moving picture comprising a plurality of 
scenes, each scene comprising a plurality of cuts and , — . . — . 
each cut comprising a plurality of frame images; detect- ' — ' ' — I 
ing means for detecting change points of said moving [~| Q 

picture and dividing said moving picture into said 

scenes and cuts; selecting means for extracting a still ™323 H-^312 

image representing each of said scenes and cuts; a 
memory for storing said related information and the rep- 
resentative still images of said scenes and cuts; means 321 1 h^h i jpgH j i — I I — I brsk-306-1 
for designating said representative still images of said 
scenes and cuts to edit said moving picture; means for 
determining a hierarchical structure, said hierarchical 
structure comprising a plurality of layers in accordance 
with said scenes and cuts; means for displaying said 
hierarchical structure and said related information; and 

a controller for controlling said memory and said display / enlargement s 304 

means so that said related information in relation to said 
representative still images to be displayed is displayed 
with said representative still images. 
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Description 

BACKGROUND OF THE INVENTION 

[0001 ] The present invention relates to a moving pic- s 
ture editing apparatus for use in production of such a 
moving picture program as television broadcasting pro- 
gram or video program. 

[0002] The performance of a facility for television 
broadcasting programs or video programs has conven- w 
tionally been advanced rapidly with the help of compu- 
ter. In these years, in particular, remarkable 
advancement is directed not to a video editing method 
of a video tape recorder based on repeating fast-for- 
ward and fast-reverse depending on a tape counter, but is 
to a moving picture editing apparatus for storing source 
material video pictures in an inexpensive hard disk 
which is easy to handle and save for editing of source 
material video pictures, that is, to a so-called nonlinear 
editing apparatus. 20 
[0003] Fig. 2 shows a block diagram of a nonlinear 
moving picture editing apparatus. 
[0004] The illustrated apparatus includes a scene 
change point detector 201 , a magnetic memory device 
202, a monitor 203, a frame buffer 204, a main memory 25 
205. an input device 206, a video tape recorder (VTR) 
207. a video interface 208, a picture compressor 209, 
an audio interface 210, a central processing unit (CPU) 
2 1 1 . a bus 212 and a picture size reducer 214. 
[0005] The scene change point detector 201, mag- 30 
netic memory device 202. monitor 203, frame buffer 
204, main memory 205, input device 206. VTR 207, 
audio interface 210, picture compressor 209 and picture 
size reducer 214 are connected to the CPU 21 1 through 
the bus 212. The VTR 207 is connected to the video 35 
interface 208 and audio interface 210. 
[0006] In Fig. 2, the CPU 21 1 performs data transfer 
with the respective constituent elements connected 
thereto via the bus 212, the constituent elements being 
controlled by an access signal received from the CPU 40 
21 1. An instruction to the CPU 21 1 is carried out by an 
operator with use of the input device 206 such as a 
mouse or keyboard. The VTR 207 reproduces a series 
of moving picture information containing plurality of 
scenes, sends moving picture in the reproduced infor- 45 
mation to the video interface 208, and also sends an 
audio signal in the reproduced information to the audio 
interface 210. The audio interface 210 converts the 
received audio signal to a digital signal and sends the 
digital signal to the magnetic memory device 202. The so 
magnetic memory device 202 stores therein the 
received audio information. The video interface 208 
converts the received moving picture information to 
information having a format processable by the moving 
picture editing apparatus, and sends the converted 55 
information to the scene change point detector 201, 
magnetic memory device 202, main memory 205, pic- 
ture compressor 209 and picture size reducer 214. The 
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picture compressor 209 compresses digital data of the 
reproduced moving pictures, e.g.. based on a moving 
picture experts group (MPEG) or motion joint photo- 
graphic experts group (motion JPEG) system. The mag- 
netic memory device 202 stores therein the received 
video information. The scene change point detector 201 
analyzes the received moving pictures on a frame basis 
and detects a scene change point and a cut change 
point on the basis of a degree of change in a color dis- 
tribution between frames. The picture size reducer 214 
generates a reduced moving picture (M-icon) by thin- 
ning out data of each frame into video or moving picture 
data of 80 x 60 pixels on the basis of an analysis result 
of the scene change point detector 201 . The generated 
M-icon is sent to the magnetic memory device 202 and 
stored therein. Moving picture analysis information 
(including frame numbers for scene change points or 
cut change points) indicative of a relationship between 
the M-icon and the detection result of the scene change 
point detector 201 is also sent to the magnetic memory 
202 and saved therein. The monitor 203 displays the 
moving pictures stored in the magnetic memory device 
202 on the basis of analysis information thereof to offer 
a graphical user interface (GUI) to the operator. The 
frame buffer 204 stores therein pictures to be displayed 
on the monitor 203. The main memory 205 stores 
therein various sorts of control programs for the CPU 
211. 

[0007] In this connection, the magnetic memory 
device 202 may be replaced by another recording 
medium such as a magneto-optical memory or a remote 
file connected through a network. Further, the mouse 
used as the input device 206 may be replaced by a track 
ball or another pointing device. A communication device 
215 may be a Ether Net, an inter-face, asynchronous 
transfer mode (ATM) interface, a modem, a facsimile 
machine, a telephone or another communication termi- 
nal which can perform information transfer between an 
external communication network and the nonlinear edit- 
ing apparatus. 

[0008] The operator sequentially edits the reduced 
moving picture (M-icon) on which the scene analysis 
information obtained at the scene change point detector 
201 is reflected, and constructs them into a tree struc- 
ture. 

[0009] Fig. 3 shows an example of the editing display 
screen displayed on the monitor 203. In the drawing, 
reference numeral 301 denotes a hierarchical structure 
(a tree structure) area, numeral 302 denotes an M-icon, 

303 denotes a schematic enlargement of the M-icon, 

304 denotes a source material area, and 305 denotes a 
group of various function buttons. Editing is carried out 
by constructing M-icons into such a hierarchical struc- 
ture as shown in the hierarchical structure area 301 of 
an editing screen 300. In the tree (in the hierarchical 
structure area 301), M-icons (for example M-icon 302) 
of a lowest layer are cuts which are composed a plural- 
ity of frame images, M-icons of a upper layer of said low- 
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est layer are scenes which are composed a plurality of 
cuts, and a M-icon of a uppermost layer are moving pic- 
tures which are composed a plurality of scenes. 
Namely, in the tree having a plurality of layers, an M- 
icon is in upper layer represents a combination of M- 5 
icons in a lower layer belonging to the upper layer, which 
corresponds to a chapter or section in a story. An M- 
icon in the uppermost layer represents the story The 
source material area 304 is an area in which the M- 
icons 302 prepared by the picture size reducer 214 10 
based on an analysis result of the scene change point 
detector 201 are arranged. The operator selects ones of 
the icons to be edited and constructs them into a hierar- 
chical structure in the hierarchical structure area 301. 
The various function buttons 305 are operating buttons is 
for various sorts of editing operations on the editing 
screen 300. 

[0010] Such an apparatus as mentioned above can 
lighten operator's burden. 

[001 1] More in detail, since editing can be realized on 20 
a scene or cut (or clip) basis, story making can be facil- 
itated. Further, since division of the input moving pic- 
tures into cut units can be automatically carried out, 
operator's inconvenience of manually doing so can be 
eliminated. In addition, since the moving pictures can be 25 
stored and managed in the form of a hierarchical struc- 
ture, retrieval of a necessary scene or cut (or dip) can 
be facilitated. 

[0012] Explanation will next be made by referring to 
Figs. 4 and 5 as to a specific example (information man- 30 
aging/displaying method) of a method for managing 
related video information (linked video information) in 
the prior art nonlinear editing apparatus. Figs. 4 and 5 
show related video linked information such as source 
material names, types and lengths of source material 35 
media, and comments. Reference numeral 401 denotes 
a column for indication of different sorts of source mate- 
rial icons, reference symbol 401-1 and 401-2 denote 
source icons, numeral 402 denotes a column for indica- 
tion of names of the source materials, 403 denotes a 40 
column for indication of a distinction between video and 
audio, 404 denotes a column for indication of time dura- 
tions of the materials, 501 denotes a column for indica- 
tion of typical source material icons, 502 denotes a 
column for indication of title names, types and durations 45 
of the typical source icons, 503 denotes a column for 
indication of comments of the materials for their editing, 
and 504 denotes a column for indication of labels of the 
source materials. 

[001 3] In Fig. 4, the source icons column 401 shows so 
the source icon 401-1 and source icon 401-2 indicative 
of source material types. For example, the source icon 
401-1 indicates that the source material is a broadcast- 
ing program, while the source icon 401-2 indicates that 
the source material is a scene. Other source icons ss 
include an icon indicative of one cut or clip. The source 
material name column 402 shows names of the source 
materials given in the source icons column 401 at the 



left side of the column 402. The video/audio distinction 
column 403 indicates "A1-2" when the source material 
is of an audio type, and indicates "V1" when the source 
material is of a video type. Further, the material duration 
column 404 given at the right side of the column 403 
indicates time lengths or durations of the source materi- 
als. 

[0014] in Fig. 5, the typical material icon column 501 
shows therein an audio source icon 501-1, a still image 
source icon 501-2, a moving image source icon 501-3 
accompanied by audio information, a folder icon 501-4, 
and a moving image source icon 501-5 not accompa- 
nied by audio information. The name/type/duration col- 
umn 502 given at the right side of the column 501 
indicates names, types, time lengths, etc. of the source 
materials given in the typical material icon column 501. 
For example, the column 502 at the right side of the 
audio source icon 501-1 shows "name", "moving image" 
and "time duration" of the source material in question, 
while the column 502 at the right side of the still image 
source icon 501-2 shows "name", "still image" and "time 
duration" of the source material in question. The column 
502 a the right side of the moving image source icon 
501-3 accompanied by audio information indicates or at 
the right side of the moving image source icon 501-5 not 
accompanied by audio information shows "name", 
"moving image" and "time duration" of the associated 
source materials. The column 502 at the right side of 
the folder icon 501-4 shows "folder name", "folder" and 
"number of source materials in the folder". Further, the 
comment column 503 describes editing states of the 
associated source materials or simple information nec- 
essary for the associated source materials, and the 
source material label column 504 describes label 
names of the source materials. Examples of a nonlinear 
editing apparatus are disclosed in "Media Suite Pro 1.0 
User's Guide" by Avid Technology, Inc., 1994, pages 92, 
93, 123 and 125 and in "Adobe Premiere 4.2 User 
Guide" by Adobe Systems Incorporated, 1997, pages 
50 and 55. 

[001 5] In a prior art moving picture editing apparatus 
called the nonlinear editing apparatus, information 
accompanied by video information is limited only to the 
time length of a video source material or producer's 
comment and can be controlled only so as to give uni- 
form information to respective video source materials. 
For this reason, the prior art apparatus cannot cope with 
such various types of video-accompanied information a 
plurality of layers that are different degrees of abstrac- 
tion. 

[0016] In the course of video production, a large 
amount of individual information related to video pro- 
duction is actually generated in various formats of infor- 
mation, including, for example, such input related 
information as input date, input location and input inten- 
tion, such right related information as the right of like- 
ness or portrait, the copyright on music, usable range 
and use expiration date, such video additive information 
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as film script and caption, and such edit associated 
information as edit history and time code. 
[001 7] With respect to such video accompanied infor- 
mation, tape based editing (the linear editing apparatus) 
has been so far conducted in a broadcasting station in s 
such a manner that a video producer keeps these infor- 
mation in different places in the form of a memo note or 
pad about data collection, editing and broadcasting 
memo or in the form of a word processor floppy disk. 
Further, video information per se has been stored in 10 
video tape. In this way, information has been stored in 
different media depending on the type of the informa- 
tion. Therefore information management has had to be 
separately carried out and thus relations and correla- 
tions between the information have had been ambigu- is 
ous with time passage, leading eventually to scatter and 
loss of the information. The same situation has been 
occurred in the conventional linear editing system. 

SUMMARY OF THE INVENTION 20 

[0018] It is therefore an object of the present invention 
to provide an apparatus for editing moving picture hav- 
ing an related information called an integrated informa- 
tion management system which can relate image units 2s 
of a moving picture to additive information associated 
with the image units and having various types of infor- 
mation. 

[0019] In accordance with an aspect of the present 
invention, the above object is attained by providing an 30 
apparatus for editing moving picture having related 
information thereof. This apparatus comprises input 
means for inputting said moving picture and said related 
information, said moving picture comprising a plurality 
of scenes, each scene comprising a plurality of cuts and 35 
each cut comprising a plurality of frame images; detect- 
ing means for detecting change points of said moving 
picture and dividing said moving picture into said 
scenes and cuts; selecting means for extracting a still 
image representing each of said scenes and cuts; a 40 
memory for storing said related information and the rep- 
resentative still images of said scenes and cuts; means 
for designating said representative still images of said 
scenes and cuts to edit said moving picture; means for 
determining a hierarchical structure, said hierarchical 45 
structure comprising a plurality of layers in accordance 
with said scenes and cuts; means for displaying said 
hierarchical structure and said related information; and 
a controller for controlling said memory and said display 
means so that said related information in relation to said so 
representative still images to be displayed is displayed 
with said representative still images. 
[0020] In accordance with another aspect of the 
present invention, the above object is attained by pro- 
viding a method for integratedly managing a moving pic- 55 
ture and additive information under control of a 
computer, which comprises the steps of displaying 
selected typical ones of frame images of the moving pic- 



ture on a display device in the form of a hierarchical 
structure, and displaying additive information associ- 
ated with the moving picture on the display device in the 
form of representation of degrees of abstraction differ- 
ent according to the layers of the hierarchical structure, 
as associated with the displayed typical frame images. 
[0021] In accordance with a further aspect of the 
present invention, there is provided a computer program 
product which includes a recording medium in which a 
procedure in the above integrated information manage- 
ment method is recorded in the form of a format reada- 
ble by the computer. 

[0022] In the present invention, the typical frame 
images of the moving picture are related to the respec- 
tive additive information associated with the images, the 
additive information is displayed on the screen of the 
display device as associated with the frame images, 
and also displayed in the form of representation of dif- 
ferent degrees of abstraction different according to the 
layers of the frame images associated with the additive 
information in the hierarchical structure of the moving 
picture. Therefore, the moving picture and additive infor- 
mation can be all managed one-dimensionally on the 
screen, and thus can support the editing operation of 
the moving picture so as to match editor's thinking pat- 
tern. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0023] 

Fig. 1 shows an example of an editing display 

screen in a moving picture editing apparatus in 

accordance with the present invention; 

Fig. 2 is a block diagram showing an example of a 

nonlinear moving picture editing apparatus; 

Fig. 3 shows an example of an editing display 

screen during nonlinear editing operation; 

Fig. 4 shows an example of a method for managing 

and displaying information in a prior art nonlinear 

editing apparatus; 

Fig. 5 shows another example of the method for 

managing and displaying information in the prior art 

nonlinear editing apparatus; 

Fig. 6 shows an information window for input and 

display of various sorts of information in the present 

invention; 

Fig. 7 is a flowchart showing an example of a 
method for relating video associated information to 
video data in accordance with the present inven- 
tion; 

Fig. 8 is a diagram for explaining degrees of 
abstraction for video associated information; 
Fig. 9 shows another example of the editing display 
screen in the moving picture editing apparatus in 
accordance with the present invention; 
Fig. 10 shows a further example of the editing dis- 
play screen in the moving picture editing apparatus 
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in accordance with the present invention; 

Fig. 11 is a diagram for explaining how to store 

video information in the present invention; 

Fig. 12 shows memory areas for storing therein 

related information and M-icon information associ- s 

ated therewith; 

Fig. 13 shows an input display screen for related 
information relating to input conditions; 
Fig. 14 shows correlations between input associ- 
ated information and M-icons; 10 
Fig. 15 shows another input display screen for 
related information relating to input condition; 
Figs. 16 to 21 show other examples of the informa- 
tion window respectively; 

Fig. 22 shows an exemplary hierarchical structure is 
in the present invention; 

Fig. 23 is a block diagram of an arrangement of the 

moving picture editing apparatus in accordance 

with an embodiment of the present invention; 

Figs. 24 to 29 show other examples of the informa- 20 

tion window respectively; and 

Fig. 30 shows a flowchart for explaining an example 

of how to set www page information. 

DESCRIPTION OF THE PREFERRED EMBODI- 25 
MENTS 

[0024] Prior to detailed disclosure of an apparatus for 
editing a moving picture and related information in 
accordance with an embodiment of the present inven- 30 
tion, explanation will first be made as to editors thought 
process course when the editor edits a moving picture 
and as to consideration results by the inventor therea- 
bout 

[0025] Fig. 8 is a diagram for explaining degrees of 35 
abstraction for video associated information. 

Specific Example 1 : 

[0026] A documentary program about "four seasons in 40 
a mountain" is assumed to be produced in Fig. 8. The 
program is intended to introduce mountain's four sea- 
sons. When a producer produces the program, pro- 
ducer's thinking procedure will be as follows. Already 
possessed by the producer as on-hand data are video 45 
data 801 indicating that Icicles thaw in soft sunlight and 
its melt water falls in drops", video data 802 indicating 
that "the river flows as swelled by melted snow", video 
data 803 indicating that "trees and plants in a field shoot 
out buds", video data 804 indicating that "animals so 
awake from hibernation and start to move", video data 
805 indicating that "a bird sings", video data 806 indicat- 
ing that "the woods are tinged with a bright yellowish 
green flush", video data 807 indicating that "cicadas are 
singing", video data 808 indicating "leaves with blazing ss 
autumnal tints", and video data 809 indicating that "all is 
covered with snow". The producer first of all, looks over 
these on-hand data in a general way, and plans to intro- 



duce video data 817 of "mountain's four seasons" 
sequentially from spring to winter and to compose the 
video data 817 into four volumes of "spring* video data 
813, "summer" videodata814, "autumn" video data 815 
and "winter" video data 816. The concept of dividing the 
"mountain's four seasons" video data 817 into the four 
volumes of the "spring" video data 813, "summer" video 
data 814, "autumn" video data 815 and "winter" video 
data 816 is more concrete than (lower in abstraction 
than) the concept of the "mountain's four season" video 
data 817. 

[0027] It is also assumed that the producer intends to 
divide a large volume of the "spring" video data 813 into 
three parts of "snow thaw" video data 810, "budding" 
video data 811 and "awaken from hibernation" video 
data 812. In this case, the concept of the "snow thaw" 
video data 810, "budding" video data 81 1 and "awaken 
from hibernation" video data 812 is higher in concrete- 
ness (lower in abstraction) than the concept of the 
"spring" video data 813. 

[0028] The producer further thinks of dividing the 
"snow thaw" video data 810 into two data of the video 
data 801 of "icicles thaw in soft sunlight and its melt 
water falls in drops" and the video data 802 of "the river 
flows as swelled by melted snow"; and also dividing the 
"budding" video data 803 into two data of the video data 
804 of "animals awake from hibernation and start to 
move" and the video data 805 of "a bird sings". 
[0029] In this case, the concept of the video data 80 1 
of "icicles thaw in soft sunlight and its melt water fells in 
drops" is higher in concreteness (lower in abstraction) 
than the concept of the "snow thaw" video data 810. 
Further, the major object of the aforementioned pro- 
gram to "introduce mountain's four seasons" is also one 
of pieces of important video associated information, 
which information has a level equivalent to the "moun- 
tain's four seasons" video data 817. 

Specific Example 2: 

[0030] Programs include a narration, which is also one 
of important pieces of video associated information. 
The producer thinks of inserting a narration A of "in 
spring warm sunlight,..." for the video data 801 of la- 
des thaw in soft sunlight and its melt water falls in 
drops" and of inserting a narration B of "trees and plants 
in a field...." for the video data 803 of trees and plants 
in a field shoot out buds". In this case, for example, the 
narration A is related to the video data 801 of "icicles 
thaw in soft sunlight and its melt water fells in drops", 
while the narration B is related to the video data 803 of 
"trees and plants in a field shoot out buds". In this way, 
if information is required to be related to each video data 
in a 1:1 relation, the narration information is considered 
to be information closely associated with the video and 
having the highest concreteness (lowest abstraction). 
[0031 ] In such an information management method as 
in Figs. 4 and 5, even when the narration information A 
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and B are added to the comment column 603 of source 
material editing state, for example, it is difficult to say the 
narration information A and B are related to which video. 
This also holds true for the same type of information 
such as caption and speech (words). 

Specific Example 3: 

[0032] When a person appears in a duration in a 
series of produced video pictures, information associ- 
ated with the person such as the portrait right or profile 
is also one of pieces of video associated information. 
Such person information is information applied only to 
part of the source material, but there is a possfoility that 
such information may reach a massive amount of infor- 
mation depending on the time length and type of the 
video information. This means that if it is tried to 
describe and manage information in such a manner as 
in Figs. 4 and 5, then this soon becomes impossible 
because of a screen space problem. 
[0033] The aforementioned video associated informa- 
tion column can describe surficial information such as 
the name and time length of a source material, but it has 
been difficult to describe information as associated with 
a flow of video information such as a film script. It has 
also been impossible to describe information applied 
only to part of a source material, such as profile infor- 
mation on a person who appears in only part of the 
source material. 

[0034] It is usually common practice as in the above 
specific example 1 that the producer regards and organ- 
izes a program as composed as a hierarchical structure. 
However, a concept generated in the course of produc- 
ing the program is one of pieces of important video 
associated information, and information having 4 stages 
(layers) of abstraction levels ranging from information 
having the highest abstraction level to information hav- 
ing the lowest abstraction level exist in the single pro- 
gram. It is impossible to describe and manage such 
information according to the aforementioned prior art 
information management method. 
[0035] Further, even when the above narration infor- 
mation is added to the conventional display column in 
the prior art method as in the above specific example 2, 
it is impossible to say which narration information is 
related to which video information. In this way, it is 
impossible to clearly describe and manage information 
generated along an editing flow of video information in 
the prior art method. This also holds true even for the 
same type of information as caption and speech. 
[0036] It is clear that the prior art display method 
involves a screen space problem as in the above spe- 
cific example 3. 

[0037] A system for integratedly managing a moving 
picture and related information provides means for solv- 
ing the above problem. 

[0038] An embodiment of the present invention will be 
explained with reference to Figs. 1 1 and 2, 
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[0039] Fig. 11 is a diagrammatic view for explaining 
how video and audio information are stored in a mag- 
netic memory device 202. Fig. 2 is a block diagram of an 
arrangement of a moving picture editing apparatus 
5 which has the same arrangement as already explained 
in connection with the prior art. 

[0040] Referring to Figs. 1 1 and 2, explanation will be 
made as to how video and audio information are stored 
in the magnetic memory device. First, an instruction 

io from the input device 206 causes a video signal to be 
issued from video tape already loaded in the VTR 207, 
so that a frame unit of moving picture is sent from the 
VTR 207 through the video interface 208 and bus 21 2 to 
the scene change point detector 201. The above- men- 

15 tioned video interface is one of the input means. The 
scene change point detector 201 digitizes the received 
moving picture, and analyzes color distribution informa- 
tion between images thereof to detect a scene change 
point and a cut change point. A frame number for the 

20 scene change point and a cut change point generated 
by the scene change point detector 201 is stored in the 
magnetic memory device 202 as moving picture analy- 
sis information (scene/cut change point information). At 
the same time, an M-icon is stored in the magnetic 

25 memory device 202. This M-icon, which is a miniature 
image (for example, of 80 x 60 pixels) reduced in size by 
the picture size reducer 214 according to the size of a 
GUI environment to be displayed, can be operated in 
the present editing apparatus. In general, in order to 

30 extract a predetermined frame from an encoded moving 
picture and convert it to a video signal for picture repro- 
duction or reconstruction, it is necessary to decode the 
encoded moving picture. In this connection, the CPU 
211 requires a processing time for decoding the moving 

35 pictures. When the miniature image (M-icon) is previ- 
ously stored as a high-speed display image as in the 
foregoing embodiment, the need for the decoding time 
can be eliminated, the processing time can be short- 
ened, and the contents of the image can be quickly con- 

40 firmed. The above-mentioned audio interface is also 
one of the input means. 

[0041] At the same time, the moving picture is 
encoded by the picture compressor 209 with use of an 
image compression technique and stored in the mag- 

45 netic memory device 202. This moving picture (com- 
pressed moving picture) is used, for example, when it is 
desired to see a final edited result in detail. With regard 
to audio, further, an audio output signal of the VTR 207 
is digitized through the audio interface 210 and stored in 

so the magnetic memory device 202. 

[0042] The operator instructs reading of an M-icon 
registered in the magnetic memory device 202 with use 
of such an input device 206 as a mouse or a keyboard. 
This causes the CPU 21 1 to read out scene/cut analysis 

55 information on the corresponding moving picture from 
the magnetic memory device 202, and on the basis of 
the information, to read out only the first frame of each 
scene or each cut from the M-icon and display it on the 
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monitor 203. The displayed M-icon is to be operated by 
the present editing apparatus. 

[0043] Fig. 1 is an example of an editing display 
screen using the aforementioned M-icon. Reference 
numerals in Fig. 1 includes, in addition to reference 
numerals already explained in the display screen of Fig. 
3, information existence display zones 306, 306-1, 306- 
2 and 306-3, a cut or scene M-icon 31 1 . an M-icon 312 
of a plurality of cuts or scenes, an M-icon 313 of the 
entire story, an information button 321 , an appearance 
person detecting button 322, a scanner icon 331, a FAX 
icon 332, a TEL icon 333 and a portrait right icon 341 . 
[0044] In Fig. 3, the operator organizes the M-icons 
302 into such a hierarchical structure as shown in a 
hierarchical structure display window 301 to thereby 
make a story. The editing method and system for editing 
a moving picture by using the editing display screen as 
shown in Fig. 3 are disclosed, for example, in the spec- 
ification of co-pending patent application Serial No. 
087826,975 filed on April 9 ( 1997. The disclosures of 
this co-pending application are incorporated herein by 
reference. 

[0045] This hierarchical structure editing system has 
not only an advantage that the operator can visuaHy 
grasp the video structure easily and can be easily make 
a story, but also another advantage that this editing sys- 
tem can present an environment optimum for efficient 
management of video related information having a plu- 
rality of layers that are different degrees of abstraction. 
Explanation will be directed to how to manage informa- 
tion, e.g., in the 3 specific editing examples already 
explained in connection with Fig. 8. 

Specific Example 1 : 

[0046] When a documentary program is prepared by 
the present moving picture editing apparatus, the then 
display screen is as shown in Fig. 10. Fig. 10 is another 
example of the editing display screen, in which 1001 to 
1005, 1011 to 1013, 1021. 1022 and 1031 denote M- 
icons and the other reference numerals are the same as 
those in Fig. 1. The M-icons 1001 to 1005 indicates 
video data per se. More specifically, the M-icon 1001 
indicates the video data 801 of "icicles thaw in soft sun- 
light and its melt water falls in drops", the M-icon 1002 
indicates the video data 802 of "the river flows as 
swelled by melted snow", the M-icon 1003 indicates the 
video data 803 of Trees and plants in a field shoot out 
buds", the M-icon 1004 indicates the video data 804 of 
"animals awake from hibernation and start to move", 
and the M-icon 1005 indicates the video data 805 of "a 
bird sings". Other M-icons not illustrated in the lower- 
most layer of the hierarchical structure are also similar 
to the above. Information having lowest one of degrees 
of abstraction is made to be related to such M-icons in 
the lowermost layer. For example, the explanation infor- 
mation of "icicles thaw in soft sunlight and its melt water 
falls in drops" is related to the M-icon 1001. and the 



explanation information of "the river flows as swelled by 
melted snow" is related to the M-icon 1002. 
[0047] In a layer higher by one layer, the M-icons 1 00 1 
and 1002 are coupled to form the M-icon 101 1. The M- 

5 icon 1011 shows one scene consisting of the video data 
801 of "icicles thaw in soft sunlight and its melt water 
falls in drops" and the video data 802 of The river flows 
as swelled by melted snow". The M-icon 101 1 is made 
to be related to the aforementioned "snow thaw" expla- 

10 nation information. Similarly, the M-icon 1012 is made to 
be related to the "budding" explanation information, 
while the M-icon 1013 is to be related to the "awaken 
from hibernation" explanation information. 
[0048] In a layer even higher by one layer, the M-icons 

15 1011, 1012 and 1013 are coupled to form the M-icon 
1021. This M-icon 1021 shows one "spring" scene con- 
sisting of the three scene parts of "snow thaw", "bud- 
ding" and "awaken from hibernation". The M-icon 1021 
is made to be related to the "spring" explanation infor- 

20 mation. Similarly, the M-icon 1 022 is made to be related 
to the "summer" explanation information. 
[0049] The M-icon 1031 in the uppermost layer shows 
the entire program. The M-icon 1031 is made to be 
related to the "mountain's four seasons" explanation 

25 information or such object information as "to introduce 
mountain's four seasons". 

Specific Example 2: 

30 [0050] For example, when narration information of "in 
spring warm sunlight,..." is made to be related to the M- 
icon 1001 in the lowermost layer while narration infor- 
mation of "trees and plants in a field shoot out buds" is 
to be related to an M-icon 1003, video and narration 

35 information can be managed completely in a 1:1 rela- 
tion. 

[0051] Further, there may occur such a case that, in 
the course of video editing, a music is superimposed on 
the video as a background music (BGM) at a later stage. 

40 Such a BGM has related information about its music 
title, time length, music player and copyright When it is 
desired to relate such information to the associated 
video, BGM in many cases spans a plurality of cuts or 
scenes. For this reason, BGM is related to a layer higher 

45 than a middle layer. When explanation is made in con- 
nection with the example of Fig. 10 and a spring BGM is 
superimposed on the entire "spring" scene; the BGM 
information is related to the M-icon 1021 . 
[0052] Further, such more global information as a 

50 broadcasting schedule for the final edited program is 
related to the M-icon 1031 in the uppermost layer. 
[0053] Fig. 29 shows a simple example of the hierar- 
chical structure, in which reference numerals 1 051 and 
1 052 denote M-icons in a lower layer of the hierarchical 

55 structure, and numeral 1050 denotes an M-icon in an 
upper layer of the hierarchical structure. When the M- 
icons 1051 and 1052 are combined, the M-icon 1050 is 
formed. Fig. 30 is a flowchart for explaining the opera- 
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13 

tion of an exemplary data structure of memory areas of 
the main memory 205 for storing video related informa- 
tion and icon information as associated therewith. In the 
drawing, reference numeral 900 denotes icon informa- 
tion of the M-icon 1050, numerals 901 and 902 denote 
video related information of the M-icon 1051, 903 
denotes icon information of the M-icon 1 051 , 904 and 
905 denote video related information of the M-icon 
1051 , 906 denotes icon information of the M-icon 1052, 
907 and 908 denote video related information of the M- 
icon 1052. 

[0054] in Fig. 30, the icon information 900 of the M- 
icon 1050 located in the upper layer of the hierarchical 
structure contains an M-icon number (or the address of 
the M-icon, such as "1" for the icon information 900, "2" 
for the icon information 901, "3" for the icon information 

902 for convenience of explanation), a layer level value 
(1 , 2,... from the bottom layer), an address for the mini- 
ature moving picture image, and address of the icon 
information of the M-icon in the layer lower by one level 
and located at the left most position, an address of the 
icon information of the M-icon located in the same layer 
and at the right side by one, and an address of the first 
video related information. Contained in the first video 
related information 901 are, in addition to such informa- 
tion as "item name", "broadcasting date" and "con- 
tents", an address of the next video related information 
in the case of presence of the next video related infor- 
mation. The same holds true even for the next video 
related information 902. The icon information 903 of the 
icon 1051 located in a layer lower than the M-icon 1050 
and at the leftmost position is associated by the address 
stored in the icon information 900 of the M-icon 1 050. 
Contained in the icon information 903 are, in addition to 
"M-icon number'', "layer level value", "address of minia- 
ture moving picture display image", "address of M-icon 
in the same layer and at the right side by one" and 
"address of the first video related information", such 
information as "file name of miniature moving picture 
image" and " cut startfend points" as the M-icon infor- 
mation of the lowermost layer. The same holds true 
even for the next icon information 906 as in the icon 
information 906. The video related information 904, 
905, 907 and 908 associated with the icon information 

903 and 906 are also similar to the video related infor- 
mation 901 and 902, and thus explanation thereof is 
omitted. 

[0055] There is an M-icon 1060 (with icon information 
909) in the layer higher than the layer of the M-icon 
1050. If there is also an M-icon 1061 (with icon informa- 
tion 910) (not shown) in the same layer, then its M-icon 
number stored in the icon information 900 is also 
recorded in the icon information 909 as an address, and 
the M-icon number of the icon information 910 is stored 
as the address of the icon information of the M-icon 
located in the same layer as the icon information 900 
and at the right side by one therefrom. 
[0056] In this manner, the hierarchical structure of the 



M-icons forming the moving picture and video related 
information are stored as associated with each other in 
the form of the hierarchical structure. 
[0057] The "contents" column in the icon information 

5 of the video related information stores actual informa- 
tion as character data, or when the video related infor- 
mation is present in another file in such a data format as 
audio or video, contains address information indicative 
of the presence location of the file. Further, when the 

10 video related information is present in a file of an exter- 
nal information system, the "contents" column of the 
video related information contains a code or uniform 
resource locator (URL) for access to the file of the infor- 
mation system. 

15 [0058] In this way, the information management 
method of the present invention based on the linked 
information to the hierarchical structure enables sophis- 
ticated information management according to the 
degree of abstraction in information. 

20 [0059] When such information linking is carried out, 
for example, on the display screen of Fig. 1, a "script" 
icon and "caption" icon (not illustrated in Fig. 1) are dis- 
played in an edit associated information zone 306-1 of 
an M-icon 311 indicative of a cut or scene in a lower 

25 layer; a "music" icon, a "player" icon or an "author" icon 
indicative of a music title, player name or copyright of a 
BGM is displayed (not illustrated in Fig. 1) in an edit 
associated information zone 306-2 of an M-icon 312 
indicative of a plurality of continuous cuts or scenes in 

30 the middle layer; and an "origin" icon indicative of a 
linked broadcast history is displayed (not illustrated in 
Fig. 1) in an edit associated information zone 306-3 of 
an M-icon 313 placed in the uppermost layer and indic- 
ative of a story. 

35 [0060] In the connection, video associated informa- 
tion, which relates to the miniature moving picture itself 
and does not depend on the video data combination or 
the edited result can be related even in the source 
material area 304. 
40 [0061] The prior art moving picture editing apparatus 
can perform only uniform management over information 
having various different degrees of abstraction depend- 
ing on different video source materials. On the other 
hand, the management method of the present embodi- 
es ment based on the aforementioned linking to the hierar- 
chical structure can realize sophisticated information 
management depending on the information abstraction. 
[0062] Television broadcasting programs and video 
programs are divided into full editions containing all vtd- 
50 eos from the beginning to the very last and digest edi- 
tions of extraction of only gist. In the present editing 
apparatus, it is assumed to edit and prepare a full edi- 
tion of program. In the lower layer of the prepared hier- 
archical structure, it is judged whether each cut or 
55 scene is used for the digest edition, and digest edition 
usable information is previously given to the cut or 
scene for use in the digest edition. Further, for switching 
between the full and digest editions, are previously 
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given (not shown in Fig. 1) in the edit in the uppermost 
layer. When it is desired to actually use the digest edi- 
tion, the producer can quickly change the program to its 
digest edition only by switching the full/digest edition 
information given in the uppermost layer M-icon to the 
digest edition. In this way, when individual information 
are given in the lower layer and its management infor- 
mation is added to the upper layer, one edited result can 
have many variations for use in various types of broad- 
cast. 

[0063] Information for display of such lilm script", 
"caption", "music", "player", "copyright", "origin", "full" 
and "digest" icons as explained above on the display 
screen are stored in such memory areas as shown in 
the example of the related information boxes 801 , 802. 
etc. in Fig. 12. These icon information are automatically 
generated by the computer according to the contents of 
the related information. Information for display of the 
"full" and "digest" icons are management information at 
the time of outputting or editing specific contents in the 
related information of the M-icon image. Such manage- 
ment information include, as other information, informa- 
tion for selective allocation to Japanese or English 
narration. 

[0064] Explanation will next be made as to how vari- 
ous types of video linked information are input to which 
computer to be related to video information and as to 
how the related related information are displayed for the 
operator in the practical editing apparatus. 
[0065] An example of an information linking method in 
the present editing apparatus will be first explained. 
[0066] Explanation will be made in connection with an 
example wherein, in such a hierarchical structure as 
shown in Fig. 1 , video linked information, e.g., portrait 
right information about a person A appearing in the 
video is present for the typical scene of the M-icon 302. 
[0067] Media for telling the operator the portrait right 
information include notepad, facsimile and telephone. 
[0068] When the portrait right information is told to the 
operator in the form of notepad, there are two informa- 
tion linking methods which follow. 
[0069] In the first linking method, under a condition 
that the M-icon 302 is selected with use of the mouse, 
the mouse is clicked on the information button 321 . This 
causes appearance of such an information window as 
shown in Fig. 6 for input and display of various types of 
information on the scene on the monitor 203. In Fig. 6, 
reference numeral 600 denotes an information window 
associated with the rights, numeral 601 denotes a 
check box for activating a portrait right icon, 602 
denotes an information column, 603-1, 603-2 and 603-3 
denote tag buttons for switching from the current infor- 
mation window to another information window associ- 
ated with another information-genre, 604 denotes a 
check box for giving an emphasized indication of the 
expiration date, 605 denotes a reference button for 
referring to another linked multimedia information 
medium, 606 denotes an OK button for validating the 



input result, and 607 denotes a cancel button for invali- 
dating the input result. The information window is used 
to display a list of information items and its information 
contents. The operator writes the contents of the note- 
5 pad in the information column 602 at the side of the por- 
trait right item with use of such an input device 206 as a 
keyboard. 

[0070] When the above is viewed from the apparatus 
side, its flowchart is as shown in Fig. 7. When an inter- 
red rupt input is generated (step 100) from the input device 
206 (see Fig. 2) such as a mouse, the CPU 211 first 
checks whether or not the interrupt input results from a 
depression of the information button 321 (see Fig. 1) 
(step 101). If so as a result of the check, then the CPU 
is checks whether or not the M-icon 302 was already 
selected prior to the generation of the interrupt (step 
102). When determining that the M-icon 302 was 
already selected as a result of the check, the CPU 
causes the information window 600 (see Fig. 6) to be 
20 displayed on the monitor 203 (step 103). 

[0071 ] Next the CPU 211 waits for an input of an inter- 
rupt from such an input device 206 as a mouse or a key- 
board(step 104). Generation of the interrupt causes the 
CPU 211 to execute associated processing operations 
25 depending on the type of the interrupt (steps 105 to 1 1 1 
steps 112 to 119). 
[0072] For example, in the case of the presence of an 
input from the keyboard as the input device 206 to the 
information column 602 (step 107), the CPU 211 writes 
30 its input contents in the main memory 205 (step 113). 
[0073] When the mouse as the input device 206 is 
clicked on the check box 601 to give a check mark in the 
box (step 108), the CPU 21 1 reads out indication on/off 
information of the portrait right icon 341 from the main 
35 memory 205 (in which the indication on/off information 
is previously stored) and switches it to gain write it in the 
main memory 205 (step 1 1 4). 

[0074] When the mouse is clicked on the tag 603-1 
(step 1 1 1), the CPU 21 1 switches the information genre 
40 of the information window 600 to the associated one for 
its display (step 1 1 7). 

[0075] When the mouse of the input device 206 is 
clicked on the check box 604 for instruction of empha- 
sized or highlighted indication (step 109), the CPU 21 1 

45 reads out the indication on/off information of the portrait 
right icon 341 from the main memory 205. When the 
indication on/off information is on (step 1 15), the CPU 
reads out highlight indication on/off information of the 
portrait right icon 341 from the main memory 205 (in 

so which the highlight indication on/off information is previ- 
ously stored), switches it to write it again in the main 
memory 205 (step 119). When the mouse of the input 
device 206 is clicked on the reference button 605 (step 
110), the CPU 21 1 checks the presence or absence of 

55 these link contents. The presence of the link contents 
causes the CPU to display it on the monitor 203 (step 
116). 

[0076] When the mouse of the input device 206 is 
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clicked on the OK button 606 (step 105), the CPU 21 1 
reads out all information written in the main memory 
205 after the information window 600 is opened and 
until then from the main memory, and again stores it in 
the main memory 205 as related information of the M- 
icon 302 (step 112). And the CPU closes the informa- 
tion window 600 and releases the select state of the M- 
icon 302 to return it to the initial state (step 118). 
[0077] When the mouse is clicked on the cancel but- 
ton 607 (step 106), the CPU discards all information 
written in the main memory 205 after the information 
window 600 is opened and until then, closes the infor- 
mation window 600, and releases the select state of the 
M-icon 302 to return it to the initial state (step 1 18). 
[0078] When the mouse is clicked with the mouse 
pointer placed on the "shoot associated" tag button 603- 
2 on the screen of Fig. 6, this causes appearance of 
screen areas or boxes for input of related information of 
the "shoot associated" shown in Fig. 13 as linked to the 
M-icon. Shoot date information is entered in a box 610, 
shoot place information is entered in a box 611, and 
producer's or shooter's shoot intention is written in a box 
612. Relations between the input related information 
and M-icons are such a hierarchical structure as shown 
in Fig. 14. Buttons 615. 616 and 617 have substantially 
the same functions as the buttons 605, 606 and 607 in 
Fig. 6 respectively. 

[0079] When the operator clicks the mouse with the 
mouse pointer placed on the "edit-associated" tag but- 
ton 603-3 on the display screen of Fig. 6, this causes 
appearance of screen areas or boxes for input of "edit- 
associated" related information shown in Fig. 15 as 
. linked to M-icons. Edit instruction information is entered 
in a box 620, an editor name is entered in a box 621, 
and producer's or editor's edit intention is written in a 
box 622. Buttons 625, 626 and 627 have substantially 
the same functions as the buttons 605, 606 and 607 in 
Fig. 6 respectively. 

[0080] In this manner, individual linking is realized. 
[0081] The other linking method when portrait infor- 
mation is given in the form of a notepad is to use a scan- 
ner. The operator use the scanner to input image data 
on the notepad into a computer. This causes the input 
image data to be displayed as the scanner icon 331 in 
the source material area 304 of Fig. 1. The operator 
drugs the scanner icon 331 and drops it in the M-icon 
302 with use of the mouse. The computer applies a 
character recognition technique to the dropped image 
data to automatically judge that the data is a description 
on the portrait right, and inputs it in the portrait right 
item. In this manner, the portrait right information is 
linked to the M-icon 302. 

[0082] If the information is input from a facsimile 
machine directly into the computer, then the information 
is similarly indicated in the source material area as the 
FAX icon 332. When the operator drags the FAX icon 
and drops it in the M-icon 302, this causes the data to 
be input to the portrait right item. 



[0083] When related information comes from a tele- 
phone, its voice is input to the computer. The input voice 
data is indicated in the source material area 304 as the 
TEL icon 333, followed by such operations as men- 

5 tioned above. 

[0084] Although how to input multimedia information 
has been explained, as an example, in connection with 
multimedia icons such as the scanner icon 331, FAX 
icon 332 and TEL icon 333 as typical ones in Fig. 1, it 

w will be obvious that another input method may be 
employed. Further, the multimedia icons may be dis- 
played in an area other than the source material area 
304 or in another opened window, and one type of 
icons, e.g., a plurality of scanner icons may be 

15 employed. Furthermore, when it is impossible to display 
the icons on a single screen, the screen may be 
switched by a screen scrolling function or page chang- 
ing function. 

[0085] Fig. 9 is a view for explaining a case where it is 

20 desired to link identical portrait right information to a 
series of a plurality of scenes. Reference numerals 701 
to 706 denote M-icons indicative of cuts or scenes 
respectively, numeral 707 denotes an M-icon indicative 
of a collection of the M-icons 701 to 704, and 708 

25 denotes an M-icon indicative of a collection of all the M- 
icons 701 to 706 on the screen. When it is desired to link 
specific ones of the M-icons 701 to 706 with a specific 
information, it is only required to select a plurality of M- 
icons from the M-icons 701 to 706 and to subject the 

30 selected icons to the similar operations to the above. 
When M-icons to be selected are the icons 701 to 704, 
linking may be realized by selecting its upper-layer M- 
( icon 707, or by linking M-icons located still in the source 
material area 304 and not edited yet. 

35 [0086] The above procedure is when it is desired to 
link the portrait right information to the specific M-icon 
302. When it is difficult to know which scene where the 
person A appears at a first glance or it is troublesome to 
select a plurality of M-icons because of a large amount 

40 of M-icons, there is a convenient linking method, which 
will be explained in connection with Fig. 9 as an exam- 
ple. Assuming that the person A appears only in the M- 
icons 701 and 703 in the scenes of the M-icons 701 to 
706, then the operator first selects the M-icon 708 in the 

45 uppermost layer indicative of the entire story, and clicks 
on the appearance person detecting button 322. The M- 
icon to be selected is not limited only to one in the 
uppermost layer. Thus, when a duration where the per- 
son A appears is previously known, the M-icon in its 

50 upper layer, e.g., the M-icon 707 may be selected. That 
is, video data belonging to the M-icon 708, i.e., video 
data of the M-icons 701 to 706 are displayed in the form 
of a list spaced at suitable intervals. The operator 
searches the icons for one where the person A appears 

55 and issues a instruction indicating that the icon has the 
person A with use of mouse's click. And for starting the 
detection, the operator clicks on the appearance person 
detecting button 322. 
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[0087] The editing apparatus extracts features of the 
instructed person A, e.g., skin color, hair color, height, 
etc. therefrom, and detect, on the basis of the extracted 
features, the person A from the videos of the M-icons 
701 to 706. This function of detecting videos including 5 
the specific target person can use such a technique as 
disclosed in, e.g., JP-A-6-223179 or JP-A-9-179880. 
This technique is based on the image recognition tech- 
nique similarly to the aforementioned scene change 
point detecting method. Completion of the detecting 10 
operation causes the icon indicative of an abstract of 
the obtained detected information, that is, the M-icon 
709 having the person A detected therefrom to be cre- 
ated and displayed. When the M-icon 709 is selected 
with the mouse click and such information as, for exam- 15 
pie, portrait right information is linked thereto by the 
aforementioned keyboard input method or by a drag & 
drop method; the information is automatically distrib- 
uted to all the M-icons 701 and 704 containing the per- 
son A and thus the linking is finished at a time. This 20 
function enables various types of linking, for example, 
linking of product manufacturer's information of routine 
voice often used in a commercial message such as a 
product catch phrase (detected by the voice recognition 
technique) broadcasted together with its video in a com- 2 s 
mercial message to the detected scenes, or linking of 
sea information of specific effect sound such as sound 
of sea waves detected by the voice recognition tech- 
nique to the detected scenes. 

[0088] in this way. the video analyzed information 30 
obtained by the video analysis technique is effectively 
utilized to realize more flexible linking of the video infor- 
mation to video associated information. 
[0089] Explanation will next be made as to how to dis- 
play the linked information. 35 
[0090] In the above method, the M-icon 302 linked to 
the portrait right information has an information zone 
306 in a lower part of its miniature moving picture, in 
which a small portrait right icon 341 is displayed. The 
portrait right information indicates that the portrait right 40 
information is linked to the scene. As a result, it 
becomes quite clear that what kind of information is 
attached to the scene and thus this draws attention of 
the editor who wants to use the scene. This icon may 
use, in addition to character, symbol, illustration, pic- 45 
ture, etc., as a matter of course. In the scene having 
video expiration date information attached thereto, 
when the date is already expired or comes close 
thereto, it is indicated by the button using a featured 
color or frame. This is highly effective to avoid the scene so 
having the due expiration date from its erroneous use. 
In this way, by emphasizing it by changing the color, 
more detailed information can be given to the display. 
This can also be realized by changing the size or using 
a decorated frame. 55 
[0091] When the operator wants to see the contents 
of information linked to the typical scene such as the M- 
icon 302 in Fig. 1 , the operator clicks the mouse on the 



information button 321 (see Fig. 1) under a condition 
that the M-icon 302 is already selected with use of the 
icon. This causes appearance of such an information 
window 600 as shown in Fig. 6. The information window 
600 is divided into a plurality of subwindows for different 
genres such as shoot associated information, rights 
associated information, video related information and 
edit associated information. Change-over between the 
subwindows is carried out by selecting one of the tag 
buttons 603-1 to 603-3. The turning ON of OFF of the 
indication of the aforementioned portrait right icon 341 
is carried out by checking or not checking the check box 
601. In the above linking method, with regard to the 
items of information received from an electronic 
medium such as a facsimile machine or a scanner, the 
reference button 605 next to the information column 602 
is displayed. When the operator clicks the mouse on the 
reference button 605, image data input by the facsimile 
machine or scanner is displayed on the monitor 203. 
Also displayed on the monitor is such information as 
facsimile number, etc. of the sender. Further, voice 
information obtained from a telephone is also similarly 
reproduced by clicking on the reference button 605. In 
this way, the present invention is equipped with an easy 
calling method from the information window to each 
multimedia information. 

[0092] Further, for example, when profile information 
about the appearance person is present in an internet 
www page, this reference button can also be used when 
the operator enters its front page address in the person 
profile item 608 to refer to the front page. 
[0093] When the operator wants to search for a group 
of the linked information pieces, it can be realized in 
such a manner as will be explained below. The operator 
first selects an M-icon in the uppermost layer of the hier- 
archical structure located higher than a layer containing 
an icon for which he wants to search in the display 
screen of Fig. 1. If necessary, a plurality of the upper- 
most layer M-icons may be selected. Under this condi- 
tion, when the operator clicks on a search button 323 in 
a group of various kinds of function buttons 305, a win- 
dow for entrance of searching conditions appears. The 
searching conditions include such a key word condition 
as, e.g., name or date, and such a quantitative condition 
of a feature extracted based on an application of the 
above image recognition technique as, e.g.. a woman or 
women with brownish hair. Or when such information as 
camera work information is already linked, a condition 
of zooming-in video can be used as the searching con- 
dition. After entering a search condition, the operator 
dicks on the search button, which starts searching 
operation. After completion of the search, its searched 
result is displayed. In the case of presence of the 
searched result, the corresponding M-icon is high- 
lighted with color. In the case of absence of the 
searched result, a message saying its failure is dis- 
played. 

[0094] When the operator selects the FAX icon 332 as 
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the typical multimedia icon and clicks on the search but- 
ton 323, all the M-icons associated with the facsimile 
information are highlighted. This holds true even for 
another scan icon. 

[0095] The search in this case corresponds to a s 
reverse of the calling of the multimedia information from 
the video information by the reference button 605 (see 
Fig. 6). That is, bi-directional calling method is realized. 
[0096] Further, even when the operator selects the 
person icon 709 as the typical video analysis informa- 10 
tion button obtained based on the application of the 
video analysis technique, all the M-icons linked thereto 
are similarly highlighted, enabling quick searching oper- 
ation. 

[0097] Next explanation will be made as to the infor- is 
mation input method (fixed format system) of Fig. 6 and 
another information input method (which will be referred 
to as the free-format system, hereinafter). 
[0098] Video associated information are divided into 
many types of genres. For the purpose of managing 20 
such information, it is general to dividing such informa- 
tion into similar groups. However, such information 
grouping varies from information user to information 
user or from production site to production site. For 
example, in a certain production site, a producer wishes 25 
to collect rights associated information as one group for 
easy seeing of them as a list, because he considers the 
rights associated information to be important from the 
business viewpoint. In another production site, a pro- 
ducer may want to collect only use expiration date infor- 30 
mation on necessary videos as one group since all the 
rights associated information are not necessary. Fur- 
ther, even the types per se of necessary information 
may vary from user to user or from production site to 
production site. Furthermore, video associated informa- 35 
tion are not always present equally for all source videos. 
For example, portrait rights will not be present for all 
video materials and some video materials have no por- 
trait rights. In this way, video associated information is, 
in other words, a small amount and wide range of types 40 
of information. 

[0099] In the fixed format system, information columns 
are previously prepared equally for all source materials. 
In order to cope with a diverse range of types of infor- 
mation, rt is necessary in this system to install a multi- 45 
plicity of information columns. This system presents no 
problem so long as information are attached to or writ- 
ten in all the multiplicity of information columns. In 
actual, however, most video associated information are, 
in many cases, a small amount and many types of infor- so 
mation. In other words, only part of the installed infor- 
mation columns is used and many of the information 
columns are blanked. That is, a needless GUI space 
must be prepared for such blank columns. 
[0100] Further, an increase in the GUI space involves 55 
another problem that the operator cannot grasp the 
presence of information. In addition, even when a plural- 
ity of information pieces of an identical type are present, 



H is hard to manage them. 

[0101] This will be explained by referring to Fig. 5 
already explained in the prior art. In Fig. 5, there are 3 
types of typical information columns, that is, the name 
column 502, comment column 503 and label column 
504. The name column is used for entry and display of 
the names or type of source materials or folder, time 
lengths of the source materials or the number of source 
materials in the folder. This types of information are 
always present for any source materials. As will be seen 
from Fig. 5, such information are written in all the rows 
of the name column, and thus there is no needless 
space from the viewpoint of GUI space. However, when 
attention is directed to the comment column next to the 
name column, the rows of the comment column for the 
audio source icon 501-1 and folder icon 501-4 are 
blanked. Similarly, some rows of the label column are 
also blanked. These blanks are a needless part of the 
GUI space. 

[0102] In this example, since the number of source 
materials is 5 and the number of information types is 3, 
such blanks will not cause a serious problem. However, 
when the number of source materials or the number of 
information types is increased, a wasteful part in the 
GUI space involves a significant problem. For the mov- 
ing image source icon 501-5, further, such comment 
information as "Cut before she swings back 0:00:02:07" 
is already placed in the information column 503. When 
the operator wishes to add other comment information 
to the column 503, however, the number of lines in the 
entry column must be increased. An increase in the 
number of lines also causes increase of the numbers of 
lines in the name column and label column, thus result- 
ing in wasteful use of the GUI space. 
[01 03] GU I problems will be discussed above. 
[0104] Explanation will be directed first to a handling 
problem. 

[0105] In a moving picture editing site, it is often that 
such source material information as the name and time 
length of a shot video source material, such rights infor- 
mation as use expiration date and usable range, and 
such edit instruction information as how the video 
source material is edited and the final time length of 
finally processed video data are given to the operator in 
the form of a manually-written notepad from shoot infor- 
mation of the shot video source material such as shot 
date and shot location. As a computer-supported edit- 
ing apparatus is introduced in a moving picture editing 
site, such a notepad has been changed from its physical 
notepad to a text file form readable by a computer. 
There is a case where a description of such related 
information already exists in the form of a text file and it 
is desired to input the text file into the editing apparatus 
through a floppy disk or internet or intranet and to divide 
it and add it according to the information type. In this 
case, according to the fixed format system, the operator 
must conduct such operational steps as to open the text 
file, find corresponding contents therefrom, cut out its 
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character string with use of a mouse, copy it on a mem- 
ory, set a corresponding information column in its enter- 
able state by the aforementioned method, and then 
pasting it into the column. 

[0106] Further, when the operator wants to shift it to 
an item belonging to the entered information, he must 
perform such steps as to re-enter the information in an 
information column as its shift destination and then to 
erase the information at the shift originator. In this way, 
the operational steps are troublesome. 
[0107] Explanation will next be directed to an opera- 
tion problem. 

[0108] As has been explained above, in the moving 
picture editing site, various sorts of information are gen- 
erated, and there occurs such a case that some of the 
information cannot be suitably divided into prepared 
information classifications. Further, in some cases, a 
plurality of information pieces are combined and trans- 
mitted in the form of a single sentence. There are cases 
where the operator wants to link such combined infor- 
mation as it is or to temporarily enter rt with intent to later 
allocate the information pieces to respective data. 
[0109] In the above case, the fixed format system is 
inconvenient because the system always requires that 
the generated information be subdivided and linked. 
[01 1 0] Explanation will then be made below as to such 
a system that, in the above case, can flexibly input and 
display a small amount and many types of video associ- 
ated information. 

[011 1] With respect to the GUI space problem, first of 
all, the present invention can eliminate the need for pre- 
viously preparing a pair of information item and column 
as in the prior art system, and can take such a form as 
to enter any format (such as a list or sentence) of infor- 
mation in a general information column and then to 
paste a tag button of an information item corresponding 
to the input information. In other words, a necessary 
GUI space is allocated to a necessary location and any 
unnecessary GUI space for non-existing information is 
not provided. Further, since necessary information is 
arranged to be displayed quickly by means of a sort but- 
ton, which facilitates confirmation of the information. 
Even when a plurality of information pieces of an identi- 
cal type are present for an identical source material, an 
information item tag can be pasted a plurality of times 
for the respective information pieces, thus enabling 
easy and flexible handling of information. 
[0112] With respect to the handling problem, in the 
present invention, when related information is given in 
the form of a test file, an enter button to be explained 
later is used so as to be able to enter the information as 
it is into the moving picture editing apparatus. Accord- 
ingly, it becomes unnecessary to conduct troublesome 
operating steps as in the prior art. Further, since the 
operator can examine the contents of the entered infor- 
mation, he can perform information input or addition as 
if he proofread a text. In addition, since modification of 
already added information is realized in such one action 



as drag & paste a new information item tag on it, the 
handling procedure can be simplified. 
[0113] With regard to the operation problem, since 
information columns for entry of any format (such as a 

5 list or sentence) of formation as well as information item 
tags prepared in the form of a hierarchical structure are 
employed, the need for previously dividing and inputting 
information as in the fixed format system can be elimi- 
nated, and thus it becomes possible to add the informa- 

w tion roughly as the first step at the current stage and to 
later allocate it. The preparation of the information items 
in a hierarchical structure is because, when related 
information is not suitably or closely applied to one of 
the prepared information items, an information item tag 

is in a layer higher than the layer containing the informa- 
tion items and containing the concept of the related 
information can be used, enabling flexible and quick 
handling thereof. In addition, the tree-structured 
arrangement of the information item tags has an addi- 

20 tionaJ advantage that a desired tag button can be easily 
searched. 

[0114] Addition of a information type only requires 
addition of its information item tag. Even the hierarchical 
structure of the information item tags can also be set. 

25 Accordingly the present invention can flexibly cope with 
any application of the moving picture editing site in use. 
[01 15] Explanation will be made in detail as to how to 
input and display video associated information based 
on the free format system, by referring to specific 

30 embodiments which follow. 

[01 1 6] Steps until an information window is called are 
substantially the same as those in the fixed format sys- 
tem. The called window is as shown in Figs. 16 to 21. 
These drawings show views for explaining how to input 

35 and display a free format type of video associated infor- 
mation on the information window. In the drawings, ref- 
erence numeral 100 denotes an information window, 
numeral 101 denotes information item tags for various 
types of video associated information, 102 denotes a 

40 tag pasting area for pasting of an information item tag, 
103 denotes an information column for entry of any for- 
mat, 104 denotes a scroll bar for setting of an indication 
range of the information column, 105 denotes a change- 
over button for switching of an information item tag to be 

45 displayed, 1 06 denotes an importance setting button for 
setting an information importance or weight, 107 
denotes a sort button for instructing the rearrangement 
of the information indicated in the information column, 
108 denotes an input button for instructing an input of 

so contents of text file, 606 denotes an OK button for vali- 
dating the input result, and 607 denotes a cancel button 
for invalidating the input result. 

[0117] In Fig. 16, with the information window 100 
called, the operator first enters transmitted information 
55 with use of such an input device 206 as a keyboard. In 
this example, the operator enters Videos having the 
person A appearing therein can be used until December 
31, 1998\ This causes appearance of such a display 
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screen as shown in Fig. 1 7. The operator next enters a 
return at the head and tail of a character string which he 
wants to use as information in the entered sentence 
with use of the input device 206 to extract the character 
string from the entire sentence. In the present example, 
the operator enters a return at the head and tail of the 
character string of "December 31, 1998". This causes 
appearance of such a screen as shown in Fig. 18. As 
will be seen from Fig. 18, extraction statement lines 1 1 1 
are indicated at upper and lower sides of the extracted 
character string for clearly showing that the character 
string was extracted. The operator then selects a corre- 
sponding information item tag from a group of tag but- 
tons of the information item tags 1 01 . These information 
item tags are stored in the form of such a hierarchical 
structure as shown in Fig. 22. Fig. 22 shows an example 
of the hierarchical structure of the information item tags 
in the present invention. The operator switches to an 
information item tag to be displayed, by clicking the 
mouse on the change-over button 1 05 to move between 
the layers. In the present example, the operator clicks 
the mouse on the information item tag 101 of "rights 
information" in Fig. 1 8 to put the tag in its selected state, 
and then clicks the mouse on a downward arrow part of 
the change-over button 105. This causes the informa- 
tion item tag buttons 101 of "rights information", "shoot 
information", "source material information" and "edit 
information" to be changed to tag buttons 1 12 of "por- 
trait right information" and "copyright information" in a 
layer lower than the layer of "rights information" in the 
hierarchical structure of Fig. 22, as shown in Fig. 19. 
The operator further puts the information item tag 1 1 2 of 
the "portrait right information" in its selected state, and 
similarly clicks the mouse on the downward arrow part 
of the change-over button 105. This, as shown in Fig. 
20, causes appearance of information item tags 1 13 of 
"portrait/name" indicative of "portrait right holders 
name", of "portrait/expiration date" indicative of "use 
expiration date" and of "portrait/usable range" indicative 
of "portrait usable range" in a layer lower than the layer 
of "portrait right information" of Fig. 22. 
[0118] The operator drags the information item tag 
1 1 3 of "portrait/name" into the tag pasting area 1 02 at a 
position at the direct left side of the above extracted 
character string. This, as shown in Fig. 21, causes 
appearance of a copy 114 of the dragged information 
item tag 113 at the left side of the desired character 
string. 

[0119] Further, when it is desired to link a character 
string "person A" in the already-entered sentence of 
"videos having the person A appearing therein can be 
used until December 31, 1998" as portrait right holder 
name information, this can be realized by the operator 
extracting the character string from the sentence, 
selecting the information item tag 113 of "por- 
traitfname", and conducting drag & drop operation over 
it in the same manner as in the above. 
[0120] When the number of pieces of information 



exceeds a predetermined value and the information 
cannot be displayed in a single display screen, the dis- 
play range is scrolled by operating the scroll bar 104 
with use of the mouse. Even when it is impossible to dis- 

5 play the tag buttons 101 , 112 and 1 13 in the tag button 
area 109 at a time, the display range can be scrolled by 
operating a scroll bar (not shown). The hierarchical 
structure of Fig. 22 shows an example of necessary 
video associated information and partly omits parts 

io required for the explanation therein. 

[0121] With a given information item tag selected, 
operator's click of the mouse on the importance setting 
button 106 enables setting of its importance of the asso- 
ciated information. When the information item tag is set 

15 to have a high importance, the color, font, contour, etc. 
of the tag is changed, whereby the operator can see its 
importance at a first glance. 

[0122] The sort button 107 is used to change the dis- 
play order of already added information. Operator's click 

20 on the sort button 107 to issue an instruction to arrange 
the information in an order of importance, this causes 
the information to be displayed as arranged from the 
upper side of the screen to the lower side thereof 
according to the information importance. Operator's 

25 instruction to arrange and display rights-associated 
information alone causes only rights-associated infor- 
mation to be found from the added information and dis- 
played. This is a means for effectively finding important 
and/or specific information. 

30 [01 23] The operator has entered the first sentence of 
Videos having the person A appearing therein can be 
used until December 31 , 1998" from the keyboard in the 
above example. However, when the sentence is previ- 
ously given in the form of a text file, the operator clicks 

35 the mouse on the input button 108 to issue an instruc- 
tion to input the text file. This results in that the text 
appears on the information column 103. 
[01 24] When it is desired to change the information 
item of the already added information, this can be real- 

40 ized by dragging a new information item tag and drops it 
on the information item tag to be changed. 
[0125] When it is desired to invalidate the already 
added information, the operator drags the information 
item tag and drops it into an area outside the tag pasting 

45 area 102. 

[0126] Attributes such as the number of information 
item tags and item contents and the hierarchical struc- 
ture can be arbitrarily set on the moving picture editing 
apparatus by a user on the production spot. 

so [0127] Finally, when it is desired to determine the 
associated information as valid information and close 
the window, the operator clicks the mouse on the OK 
button 606. When it is desired to determine the associ- 
ated information as invalid information and close the 

55 window, the operator s click the mouse on the cancel 
button 607. 

[01 28] At the first stage, there may be considered var- 
ious types of sorting methods. For example, it is consid- 
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ered that it is desired not to finely divide the sentence of 
"videos having the person A appearing therein can be 
used until December 31, 1998" but to link it as a single 
piece of information, or that it is desired to roughly add 
information as a first step and later to finely add it. In 
such a case, it is also possible to regard the entire sen- 
tence as a single extraction and to temporarily add the 
information item tag 101 as an information item tag in 
the layer higher than the layer of the information item tag 
113 to it. 

[0129] The present invention is used, further when it 
is desired to provide an idle tag button not belonging to 
any information items as an information item tag (not 
shown) to roughly add information to it as a first step 
and later to finely add thereto because of a difficulty of 
its information sorting. 

[01 30] Although the entire sentence of "videos having 
the person A appearing therein can be used until 
December 31 . 1998" has been entered in this example, 
only the part "December 31, 1998" thereof may be 
entered from the beginning when it is previously known 
to use only the use expiration information. 
[01 31 ] In the foregoing embodiment, in order to extract 
a desired part from the information character string in 
the information column 103, divide it into line parts and 
then allocate the divided parts to video information gen- 
res; the target information item tag buttons have been 
dragged and dropped. As an alternative, however, it is 
also possible to select a necessary part in the sentence 
in the information column 103 with use of the mouse, 
and to drag the selected character string onto the target 
information item tag button to realize its allocation and 
display. 

[0132] In these years, internet world (typical one of 
which is world wide web (WWW)) has spread remarka- 
bly, and diverse information has appeared in WWW 
pages and net news groups in the internet world. 
[0133] In the foregoing embodiment, further, explana- 
tion has been made that the use of the multimedia icon 
enables input of multimedia information as video related 
information through internet communication. However, 
the above method requires that information obtained 
from the internet be once input into the moving picture 
editing apparatus. Further, when the input information is 
always updated at its input destination, it is necessary to 
again input the updated information through the network 
communication. 

[0134] Since the internet is spread and networking is 
advanced, it becomes possible to offer the aforemen- 
tioned video related information in the form of a www 
page through the internet. Further, since www page for- 
mat can be commonly used for most computers having 
different operating systems (OSs), the management 
database per se of the video related information can be 
changed to the www page format. When information of 
the www page format can be referred to on the editing 
apparatus through the network, the need for needless 
works of reconnecting the apparatus to the network to 



input the information into the apparatus can be elimi- 
nated, thus increasing its editing efficiency. 
[01 35] Explanation will then be made as to an embod- 
iment of inputting www page information by referring to 

5 Figs. 23 to 29 and 9. Fig. 23 is a block diagram of an 
arrangement of the moving picture editing apparatus in 
accordance with an embodiment of the present inven- 
tion, which corresponds to the arrangement of Fig. 2 but 
is added by the communication device 215. The appara- 

10 tus is connected to another computer or network line 
through the communication device 215 to input internet 
information into the apparatus. Usable as the communi- 
cation device 215 is an Ether Net interface allowing 
information transfer between another computer or exter- 

15 nal network and the moving picture editing apparatus, 
an asynchronous transfer mode (ATM) interface, a 
modem, a facsimile machine, a telephone set or 
another communication terminal. Figs. 24 to 29 show 
other examples of the information window, in which 

20 blocks having the same functions as those in Figs. 1 6 to 
21 are denoted by the same reference numerals. In the 
drawings, reference numeral 610 denotes a www page 
information tag, numeral 61 1 denotes a www page ref- 
erence button, 612 denotes an address setting window, 

25 613 denotes an address list column, 614 denotes a 
selected address column, 615 denotes a confirmation 
button, 616 denotes an OK button, 61 7 denotes a can- 
cel button, 618 denotes a scroll bar, and 619 denotes a 
mouse cursor. 

30 [01 36] First of ail, explanation will be made as to how 
to link www page information offered in a www page for- 
mat to a miniature moving images and how to refer it. 
[01 37] As a specific example of the method for linking 
www page information to a miniature moving image, 

35 explanation will be made in connection with a case 
where information about a person (Taro) appearing in a 
cut of the M-icon 302 in Fig. 9 is present in a www page 
introducing the profile of the person as www page infor- 
mation and the www page has an address of 

40 "httpy/www.taro.or.jp", as to how to link the www page 
information to the M-icon 302. 

[0138] The operator clicks the mouse as the input 
device 206 on the M-icon 302 to put the icon in its 
selected state, and then clicks on the information button 

45 321 in the group of various function buttons 305. This 
causes appearance of the aforementioned information 
window 100 (see Fig. 24). With the information window 
100 is displayed, the operator drags the www page 
information tag 610 with use of the mouse of the input 

so device 206 and drops it into the tag pasting area 1 02. As 
a result, the www page information tag 61 0 is pasted at 
the dropped position. If there is no information descrip- 
tion in the information column 103 at a right side of the 
pasted www page information tag 610, then the address 

55 setting window 612 appears to inquire its address as 
shown in Fig. 25. 

[0139] The address list column 613 displays a list of 
addresses referred to in the past. When there is one of 
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the www page addresses being currently referred to in 
the list, the operator select it with the mouse. In this 
example, the operator selects "http^/www.taro.or.jp ,, as 
its target address. The number of addresses displaya- 
ble in the address list column 613 at a time is 6 in the 
example of Fig. 27. However, when 7 or more 
addresses are recorded in the past and the operator 
cannot see all in the address list column 613, the oper- 
ator can operate the scroll bar 618 to select target one 
of the addresses. This causes the selected address to 
appear in the selected address column 614 (see Fig. 
26). If the operator fails to find the target address of the 
www page in the address list column 613, the operator 
enters the target address in the selected address col- 
umn 614 with use of the keyboard as the input device 
206 (see Fig. 27). In this example, the operator enters 
"http i/Awww. taro.or.jp n . 

[0140] With such a condition as shown in Fig. 26 or 
27, when the operator clicks on the confirmation button 
615, the apparatus starts to access the address shown 
in the selected address column 614 through the com- 
munication device 215 and to display obtained informa- 
tion with the help of an external application program 
designed for www page. (For example, the www page of 
the selected address accessed by an external applica- 
tion program is displayed on the monitor 203.) The 
external application is, for example, an Netscape Navi- 
gator developed by Netscape Communications Corpo- 
ration, or an Internet Explorer developed by Microsoft 
Corporation. Selection of such external application pro- 
grams can be freely done by switching between the set- 
tings of the moving picture editing apparatus. 
[0141 ] When the operator confirms the contents of the 
information displayed by the www page external appli- 
cation and judges that the www page of the selected 
address offers the desired information, he clicks on the 
OK button 616 to validate the set contents, add the 
address to the list of addresses referred to in the past 
and close the address setting window 612. Further, 
operator's click on the cancel button 61 7 causes the set 
contents to be made invalid to thereby close the 
address setting window 61 2. 

[0142] As has been mentioned above, in this example, 
the operator enters the www page address of 
"httpV/www. taro.or.jp" in the information window 100 as 
www page information. 

[0143] Further, when the operator drops the www 
page information tag 610 in the tag pasting area 102 
and a description of the www page address is given at a 
corresponding position in the information column 103, 
the description of the www page address is set as an 
address. 

[0144] . This operation as viewed from the side of the 
apparatus will be explained as a flow of operation with 
use of flowcharts of Figs. 30 and 28 showing an exam- 
ple of how to set the www page information. 
[0145] The operator first selects a desired M-icon 
(such as the M-icon 302) in the hierarchical structure 



area 301 on the editing screen 300 of Fig. 1 and clicks 
on the information button 321. This causes the CPU to 
display the information window 100 (see Fig. 24) on the 
monitor 203 (step 901) and to proceed to a step 902. 

5 Under this condition, the CPU waits for an interrupt 
input from the mouse or keyboard as the input device 
206. When accepting the interrupt input, the CPU pro- 
ceeds to a step 903 (step 902). In the step 903, the CPU 
is branched depending on whether or not the mouse 

10 was clicked on the OK button 606. That is, a click on the 
OK button causes the CPU to go to a step 904, while no 
click on the OK button causes the CPU to go to a step 
905. In the step 904, the CPU validates the set contents 
and then closes the information window 100 (end of 

w processing). In the step 905, the CPU judges which of 
the information item tags 100, 112 and 113 was 
dragged by the mouse. When determining that any of 
the information item tags 100, 112 and 113 was not 
dragged by the mouse, the CPU regards the interrupt 

20 input generated at this time as invalid and returns to the 
step 902 to waits for a next interrupt input. When deter- 
mining that any one of the information item tags 100, 
1 12 and 113 was dragged by the mouse, the CPU pro- 
ceeds to a step 906 (step 905). In the next step 906. the 

25 CPU judges whether or not any of the information item 
tags 1 00, 1 12 and 113 was dragged into the tag pasting 
area 1 02, and the CPU is put in a loop condition until the 
CPU determines that the any of the information item 
tags 100, 112 and 113 was dropped. At the time of 

30 determining that any of the information item tags 100, 
1 1 2 and 1 1 3 was dropped, the CPU goes out of the loop 
and proceeds to a next step 907 (step 906). In the step 
907, the CPU judges whether or not the dropped posi- 
tion of the information item tag is within the tag pasting 

35 area 102. When determining that the dropped position 
is out of the tag pasting area 102, the CPU regards the 
interrupt input at this time as invalid and returns to the 
step 902 to wart of a next interrupt input When deter- 
mining that the dropped position is within the tag pasting 

40 area 102, the CPU proceeds to a step 908 (step 907). In 
the step 908, the CPU displays the information item tag 
dropped within the tag pasting area 102 in the step 907 
as pasted within the tag pasting area 102, and goes to 
a step 909. In the step 909, the CPU judges whether or 

45 not the information item tag pasted within the tag past- 
ing area 102 is the www page information tag 610. If the 
information item tag is not the www page information 
tag 610, then the CPU returns to the step 902 to wait for 
a next interrupt input. When determining that the infor- 

50 mation item tag is the www page information tag 610, 
the CPU proceeds to a step 910 (step 909). In the step 
910, the CPU judges whether or not some information is 
already described in the information column 103 at a 
position corresponding to the www page information tag 

55 610 pasted within the tag pasting area 102. If some 
information is described, then the CPU proceeds to a 
step 91 1. If not, then the CPU proceeds to a step 912 
(step 910). In the step 91 1, the CPU stores its descrip- 
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tion contents and returns to the step 902 to wait for a 
next interrupt input. When determining in the step 910 
some information is not described, the CPU goes to the 
step 912 (step 91 1). In the step 912, the CPU displays 
the address setting window 61 2 on the monitor 203 and 5 
goes to a step 913 in Fig. 28 (step 912). 
[0146J In the step 91 3, the CPU waits for an interrupt 
input from such an input device 206 as the mouse or 
keyboard. Generation of an interrupt input causes the 
CPU to proceed to a step 914 (step 913). In the step 10 
914, the CPU judges whether or not an address was 
selected from the address list column 613 by the 
mouse. When an address was selected, the CPU goes 
to a step 915. When an address was not selected, the 
CPU goes to a step 916 (step 914). In the step 915, the 15 
CPU displays the selected address in the selected 
address column 614 and returns to the step 913 (step 
915). In the step 916, the CPU judges whether or not 
the address was input directly from the keyboard into 
the selected address column 614. When determining 20 
that the address was input directly into the selected 
address column 614, the CPU proceeds to a step 917. 
When determining that no address is input to the 
selected address column 614, the CPU proceeds to a 
step 918 (step 916). In the step 918, the CPU judges 25 
whether or not the mouse was clicked on the confirma- 
tion button 615. When determining that the mouse was 
clicked on the confirmation button 615, the CPU goes to 
a step 919. When determining that the mouse was not 
clicked on the confirmation button 61 5, the CPU goes to 30 
a step 921 . In the step 919, the CPU judges whether or 
not an address is indicated in the selected address col- 
umn 614. When an address is indicated in the selected 
address column 614, the CPU goes to a step 920. If not 
then the CPU returns to the step 913 (step 919). In the 35 
step 920, the CPU starts an external application pro- 
gram to access the address shown in the selected 
address column 614 and to display an obtained www 
page on the monitor (step 920). 

[0147] In the next step 921 , the CPU judges whether 40 
or not the mouse was clicked on the OK button 616. 
Determination of a click on the OK button 616 causes 
the CPU to go to a step 922. Determination of no click 
on the OK button 616 causes the CPU to proceed to a 
step 925 (step 921). In the step 922, as in the step 919, 45 
the CPU judges whether or not an address is shown in 
the selected address column 614. When an address is 
given in the selected address column 61 4, the CPU pro- 
ceeds to a step 923. If not, then the CPU returns to the 
step 913 (step 922). In the step 923. the CPU stores the so 
indication contents of the selected address column 614 
as an www page address and proceeds to a step 924 
(step 923). in the step 925, the CPU judges whether or 
not the mouse was clicked on the cancel button 617. 
When determining a click of the mouse on the cancel 55 
button 617, the CPU proceeds to a step 924. When 
determining no click on the cancel button 617, the CPU 
returns to the step 913 (step 921). In the step 924, the 



CPU closes the address setting window 612 and returns 
to the step 902 in Fig. 30. 

[0148] Explanation will then be made as to how to 
refer to the www page. 

[01 49] As a result of settings already explain in Figs. 
1 , 23 to 27, 30 and 28; the information window 100 is as 
shown in Fig. 28. This means that the www page infor- 
mation of "http://www.taro.or.jp" is associated with (e.g., 
linked to) the M-icon 302 in Fig. 1. Under this condition, 
the operator first clicks the mouse on the pasted www 
page information tag 610. This causes the www page 
information tag 61 0 to be put in its selected state, which 
is clearly shown by a red frame, indicating that the tag is 
selected. Under this condition, when the operator dicks 
the mouse on the www page reference button 611, the 
CPU starts to access the set address and display 
obtained www page information under the external 
application program. (For example, the external applica- 
tion program is started and the www page of the 
selected address is displayed on the monitor 203.) Ter- 
mination of the display is carried out by terminating the 
external application program. Finally, a click of the 
mouse on the OK button 606 causes the CPU to vali- 
date the information linked on the information window 
100, store the M-icon 302 in the hierarchical structure of 
Fig. 1 and the linked information, and close the informa- 
tion window 100. 

[0150] Even when the www page information of 
"httpy/www.whale.or.jp" is associated with the M-icon 
308 in Fig. 1 , the same method as the above is carried 
out. Further, when there is www page information to be 
linked to the entire scene such as the M-icon 307 as typ- 
ical information, for example, when there is information 
which is common to all M-icons of the layer lower than 
the layer of the M-icon 307 in the hierarchical structure 
or which is high in abstraction degree; the operator 
selects the M-icon 307 to display the information win- 
dow 100 and conducts the same procedures as in the 
above. 

[0151] In such a manner as mentioned above, the 
www page information was linked to the M-icons 302, 
308 and 307 respectively. When the operator wishes to 
again see the linked information, he selects the M-icon 
linked to the desired information with use of the mouse 
to issue an instruction to display it on the information 
window 100 in the similar way to the above, the informa- 
tion linked to the selected M-icon is displayed as shown 
in Fig. 28. Under such a condition that the M-icon 307 
located in the layer higher than the layer of the M-icon 
302 or 308 is selected with the mouse, when the opera- 
tor issues an instruction to display the information win- 
dow 100, all information contained in the layers lower 
than the layer of the M-icon 307 in the hierarchical struc- 
ture are combined and displayed in the form of a list on 
the information window 100 (refer to Fig. 29). The 
method of Fig. 29 is valid when it is unclear to which the 
target information is linked. Even in the case of Fig. 29, 
a method for referring to a www page is the same as in 
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the ease of Fig. 28. 

[0152] As has been explained above, in accordance 
with the present invention, since there can be provided 
a method for displaying video information in the form of 
a hierarchical structure, video related information can s 
be finely managed in a centralized manner and loss of 
the related information can be avoided. Further, the for- 
mation of video information in a hierarchical structure 
enables fine information management according to the 
degree of abstraction of the video related information, w 
Furthermore, the operator can quickly grasp related 
information associated with a target video source mate- 
rial during the editing operation and thus can easily con- 
firm the detailed information. In addition, since its 
operating means is implemented with a simple inter- is 
face, even an unexperienced user can operate the 
apparatus. 

[0153] In accordance with the present invention, fur- 
ther, there is provided a method for accessing informa- 
tion of a www page format from the moving picture 20 
editing apparatus through the internet based on the 
structure of video information and linking the www page 
format information to the video information according to 
the structure of the video information. As a result, infor- 
mation useful to the video editor can be more easily 25 
obtained than the prior art and thus diverse editing can 
be realized and an editing efficiency can be increased. 

Claims 

30 

1. An apparatus for editing moving picture having a 
related information thereof comprising; 

input means for inputting said moving picture 
and said related information, said moving pic- 35 
ture comprising a plurality of scenes, each 
scene comprising a plurality of cuts and each 
cut comprising a plurality of frame images; 
detecting means for detecting change points of 
said moving picture and dividing said moving 40 
picture into said scenes and cuts; 
selecting means for extracting a still image rep- 
resenting each of said scenes and cuts; 
a memory for storing said related information 
and the representative still images of said 45 
scenes and cuts; 

means for designating said representative still 
images of said scenes and cuts to edit said 
moving picture; 

means for determining a hierarchical structure, so 
said hierarchical structure comprising a plural- 
ity of layers in accordance with said scenes and 
cuts; 

means for displaying said hierarchical structure 
and said related information; and 55 
a controller for controlling said memory and 
said display means so that said related infor- 
mation in relation to said representative still 



images to be displayed is displayed with said 
representative still images. 

2. An apparatus for editing moving picture according 
to claim 1, wherein said related information com- 
prise a plurality of layers that are different degrees 
of abstraction, and said display means displays 
said representative still images and said related 
information of the layer in relation to said represent- 
ative still images to be displayed. 

3. An apparatus for editing moving picture according 
to claim 1, wherein said attribute information relat- 
ing to contents of said still images and manage- 
ment information for management of said attribute 
information. 

4. An apparatus for editing moving picture according 
to claim 1, wherein said display means further dis- 
plays icons (306-1, 306-2, 306-3) indicative of the 
related information on said display unit as associ- 
ated with said representative still images. 

5. An apparatus for editing moving picture according 
to claim 4, wherein said related information include 
information of at least any of character, video and 
audio formats. 

6. An apparatus for editing moving picture according 
to claim 5, wherein said memory includes data 
areas (900-910) for storing therein said related 
information as associated with said representative 
still images. 

7. An apparatus for editing moving picture according 
to claim 6, wherein said memory includes data 
areas for storing therein information indicative of 
presence locations of said related information as 
associated with said representative still images. 

8. An apparatus for editing moving picture according 
to claim 7, wherein said controller comprises 
means for accessing the presence location of said 
related information specified in said data areas of 
said memory by an instruction from an input device 
(206), reading out said related information from the 
memory and displaying the readout related infor- 
mation by said display means. 

9. An apparatus for editing moving picture according 
to claim 7, wherein said controller comprises 
means (215, 331, 332, 333) for accessing the pres- 
ence location of an external related information 
specified in said data areas of said memory by an 
instruction from said input device through a com- 
munication path connected to said input means, 
reading out said related information from the mem- 
ory and transferring the read-out related informa- 



18 



35 



EP 0 917 148 A2 



36 



tion by said display means. 

10. An apparatus for editing moving picture according 
to claim 6, wherein said controller comprises 
means for reading out said related information from 5 
said data areas of sad memory based on an 
instruction from said input device and displaying the 
read -out related information to said display means. 

11. An apparatus for editing moving picture according 10 
to daim 1 , wherein said controller further comprises 
means for detecting a frame image containing spe- 
cific feature information and means for relating the 
related information input from said input means to 
the frame image containing said feature informa- is 
tion. 

12. An apparatus for editing moving picture according 
to claim 11, wherein said controller further com- 
prises means for searching for the frame images 20 
and the related information stored in said memory 
and extracting the frame image containing said spe- 
cific feature information therefrom. 

13. An apparatus for editing moving picture according 25 
to claim 1 , wherein said controller further comprises 
means for displaying an related information icon for 
specification of said related information and means 

for placing said related information icon on said rep- 
resentative still image with use of said input means 30 
to correspond the related information input from 
said input means to said representative still image. 

14. An apparatus for editing moving picture according 

to claim 1 3, wherein said related information is mul- 35 
timedia information. 

15. An apparatus for editing moving picture according 
to claim 13, wherein said controller further com- 
prises means for displaying the representative still 40 
image associated with said related information as 
discriminatable from the other images. 

16. An apparatus for editing moving picture according 

to claim 1 . further comprising; 45 

said designating means comprises means for 
displaying on a display unit (203) said repre- 
sentative still image; 

said determining means comprises means for so 
selecting and positioning the representative 
image (301) in said hierarchical structure on 
said display unit to editing the .moving picture. 

1 7. A method of editing moving picture having a related 55 
information thereof comprising the steps of; 

inputting said moving picture and said related 



information, said moving picture comprising a 
plurality of scenes, each scene comprising a 
plurality of cuts and each cut comprising a plu- 
rality of frame images; 

detecting change points of said moving picture 
and dividing said moving picture into said 
scenes and cuts; 

extracting a still image representing each of 
said scenes and cuts; 

storing said related information and the repre- 
sentative still images of said scenes and cuts in 
a memory; 

designating said representative still images of 
said scenes and cuts to edit said moving pic- 
ture; 

determining a hierarchical structure, said hier- 
archical structure comprising a plurality of lay- 
ers in accordance with said scenes and cuts; 
displaying said hierarchical structure of said 
representative still images and said related 
information so that said related information in 
relation to said representative still images to be 
displayed is displayed with said representative 
sttU images. 

18. A method of editing moving picture according to 
claim 1 7, wherein said related information comprise 
a plurality of layers that are different degrees of 
abstraction, said step of displaying includes the 
step of displaying said representative still images 
and said related information of the layer in relation 
to said representative still images to be displayed. 

19. A method of editing moving picture according to 
claim 17, wherein said related information include 
attribute information relating to contents of said rep- 
resentative still images and information for man- 
agement of said attribute information. 

20. A method of editing moving picture according to 
claim 17, further comprising a step of displaying 
icons (306-1, 306-2, 306-3) indicative of the related 
information on a display unit as associated with 
said representative still images. 

21. A method of editing moving picture according to 
claim 20, wherein said related information include 
information of at least any of character, video and 
audio formats. 

22. A method of editing moving picture according to 
claim 21, further comprising a step of storing 
therein said related information in data areas of 
said memory as associated with said representa- 
tive still images. 

23. A method of editing moving picture according to 
claim 22, further comprising a step of storing 
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therein information indicative of presence locations 
of said related information in data areas of said 
memory as associated with said representative still 
images. 

5 

24. A method of editing moving picture according to 
claim 23, further comprising a step of accessing the 
presence location of said related information speci- 
fied in said data areas of said memory by an 
instruction, reading out said related information to 
from the memory and oulputting the read-out 
related information to said display unit. 

25. A method of editing moving picture according to 
claim 23, further comprising a step of accessing the 15 
presence location of said external related informa- 
tion specified in said data areas of said memory by 

an instruction provided through a communication 
path connected to a computer, reading out said 
related information from the memory and transfer- 20 
ring the read-out related information to said display 
unit. 

26. A method of editing moving picture according to 
claim 22. further comprising a step of reading out 25 
said related information from said data areas of 
said memory based on an instruction and output- 
ting the read-out related information to said display 
unit. 

30 

27. A method of editing moving picture according to 
claim 1 7, further comprising a step of detecting a 
frame image containing specific feature information 
and a step of relating the inputted related informa- 
tion with the frame image containing said feature 35 
information. 

28- A method of editing moving picture according to 
claim 27, further comprising searching for the frame 
images and the related information stored in said 40 
memory and extracting the frame image containing 
said specific feature information therefrom. 

29. A method of editing moving picture according to 
claim 17, further comprising a step of displaying a 45 
related information icon for specification of said 
related information and a step for placing said 
related information icon on said representative still 
image with use of an input device to correspond the 
related information input from said input device to so 
said representative still image. 

30. A method of editing moving picture according .to 
claim 29, wherein said related information is multi- 
media information. 55 

31. A method of editing moving picture according to 
claim 29, further comprising a step of displaying the 



representative still image associated with said 
related information as discriminatable from the 
other images. 

32. A method of editing moving picture according to 
claim 17, further comprising a step of displaying on 
a display unit the representative still image of said 
moving picture, a step of selecting the representa- 
tive still image displayed on said display unit, and a 
step of editing the moving picture by positioning in 
said hierarchical structure one of the displayed rep- 
resentative still images selected by said input 
device. 

33. A computer program product comprising a compu- 
ter usable medium storing a method of editing mov- 
ing picture having a related information thereof 
comprising the steps of; 

inputting said moving picture and said related 
information, said moving picture comprising a 
plurality of scenes, each scene comprising a 
plurality of cuts and each cut comprising a plu- 
rality of frame images; 

detecting change points of said moving picture 
and dividing said moving picture into said 
scenes and cuts; 

extracting a still image representing each of 
said scenes and cuts; 

storing said related information and the repre- 
sentative still images of said scenes and cuts in 
a memory; 

designating said representative still images of 
said scenes and cuts to edit said moving pic- 
ture; 

determining a hierarchical structure, said hier- 
archical structure comprising a plurality of lay- 
ers in accordance with said scenes and cuts; 
displaying said hierarchical structure of said 
representative still images and said related 
information so that said related information in 
relation to said representative still images to be 
displayed is displayed with said representative 
still images. 
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